Neuroadaptive intelligent virtual reality learning system and method

ABSTRACT

A computer-implemented method of providing virtual reality (VR) or Augmented Reality (AR) training includes adapting the VR/AR training based on feedback on the user&#39;s biometric data, which may include electroencephalogram (EEG) data and other biometric data. Associations are determined between the biometric data and psychological/neurological factors related to learning, such as a cognitive load, attention, anxiety, and motivation. In one implementation, predictive analytics are used to adapt the VR/AR training to maintain the user with an optimal learning zone during the training.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority under 35 USC § 119(e) to U.S.Provisional Patent Application No. 62/730,436, entitled “AdaptiveIntelligent Learning System and Method Utilizing Biofeedback” and filedSep. 12, 2018, U.S. Provisional Patent Application No. 62/742,910,entitled “Network Connecting People and Headset-Based Brain-ComputerInterface” and filed Oct. 8, 2018, and U.S. Provisional PatentApplication No. 62/743,351, entitled “Predictive Model-Based System forLower On-Device Compute Power” and filed Oct. 9, 2018, each of which areincorporated herein by reference in their entirety.

FIELD OF THE INVENTION

The present disclosure generally relates to providing neuro-adaptive(“neuroadaptive”) training using virtual reality or augmented reality.

BACKGROUND

The training of employees is a concern in many industries. Virtualreality training is being explored by a variety of companies. Virtualreality offers the potential advances of providing a way to trainemployees using an immersive environment. However, virtual realitytraining is still not as effective for enterprise training applicationsas desired.

SUMMARY

The present disclosure relates to systems and methods for usingbiometric data as feedback to adapt a training session conducted byvirtual reality or augmented reality. One aspect is that predictivetechniques may be used to adapt an education training session tomaintain the user in an optimal zone for learning. In some embodiments,this includes varying the complexity of the training, the pacing, or thesequence of learning. In one embodiment, the decisions for adapting theeducational training may be based on a set of cognitive mental statemetrics derived from two or more different types of biometric data.Using different types of biometric data permits a more detailedunderstanding of how the user is responding to a training session.

It should be understood, however, that this list of features andadvantages is not all-inclusive and many additional features andadvantages are contemplated and fall within the scope of the presentdisclosure. Moreover, it should be understood that the language used inthe present disclosure has been principally selected for readability andinstructional purposes, and not to limit the scope of the subject matterdisclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated by way of example, and not by wayof limitation in the figures of the accompanying drawings in which likereference numerals are used to refer to similar elements.

FIG. 1 is a block diagram of a system for proving neuroadaptiveeducational training in accordance with an embodiment.

FIG. 2 is a block diagram illustrating a predictive engine to adapteducational training in accordance with an embodiment.

FIG. 3 is a block diagram illustrating a predictive engine to adapteducational training in accordance with an embodiment.

FIG. 4 is a flowchart of a method of performing adaptive educationaltraining in accordance with an embodiment.

FIG. 5 is a flowchart of a method of using feedback to adapt educationbased on threshold values in accordance with an embodiment.

FIG. 6 is a flowchart of a method of forming individual thresholds,group thresholds, and weighted thresholds in accordance with anembodiment.

FIG. 7 is a flowchart of a method of using a machine learning trainingto convert biometric signals into learning metrics in accordance with anembodiment.

FIG. 8 illustrates an example of processing a raw biometric data signalinto a cognitive load score in accordance with an embodiment.

FIGS. 9-10 illustrate aspects of learning.

FIGS. 11A and 11B illustrate an EEG headset on a human head inaccordance with an embodiment.

FIGS. 12A and 12B illustrate examples of an EEG headset internalcomponent in accordance with an embodiment.

FIGS. 13A and 13B illustrate an example with an on-headset alertingdevice in accordance with an embodiment.

FIGS. 14A and 14B illustrate an example of an EEG headset without activecooling in accordance with an embodiment.

FIGS. 15A and 15B illustrate examples of an EEG headset without powerstorage in accordance with an embodiment.

FIGS. 16A and 16B illustrate examples of an EEG headset with open powervents in accordance with an embodiment.

FIGS. 17A and 17B illustrate examples of an EEG headset with sensorblocks attached in a floating fashion, in accordance with an embodiment.

FIG. 18 illustrates possible EEG sensor locations in accordance with anembodiment.

FIG. 19 illustrates possible EEG sensor locations in accordance with anembodiment.

FIG. 20 illustrates a user interfacing with a computer while the headsetdata is fed back to the computer wirelessly in accordance with anembodiment.

FIG. 21 illustrates a user interfacing with a computer while the headsetdata is fed back to the computer with a wired link in accordance with anembodiment.

FIG. 22 illustrates a user interface with a computer with the headsetgiving direct feedback to the user in accordance with an embodiment.

FIG. 23 illustrates a user interfacing with non-computer materials withthe headset data being fed back to a computer wireless in accordancewith an embodiment.

FIG. 24 is a flowchart of a machine learning routine in accordance withan embodiment.

FIG. 25 is a flow chart of a functional model operation in accordancewith an embodiment.

FIG. 26 is a flowchart of a lesson in the loop in accordance with anembodiment.

FIG. 27 illustrates a soft dry electrode with a dry sensor embedded in asoft flexible material in accordance with an embodiment.

FIG. 28 illustrates an EEG headset with an on-board computer for dataprocessing in accordance with an embodiment.

FIG. 29 illustrates an EEG headset with an external computer for dataprocessing in accordance with an embodiment.

FIG. 30 illustrates communication between EEG headsets of different userin accordance with an embodiment.

FIG. 31 illustrates communication between individual EEG headsets an dother devices in a system in accordance with an embodiment.

FIG. 32 illustrates communication between individual EEG headsets an dother devices in a system in accordance with an embodiment

FIG. 33 illustrates generation of direct visual output based on apredictive engine in accordance with an embodiment.

FIG. 34 illustrates an example of how data from multiple headsets iscommunicated back to a server in accordance with an embodiment.

DETAILED DESCRIPTION

FIG. 1 is a diagram illustrating an example of a system to provideneuroadaptive virtual reality (VR) or augmented reality (AR) training.As an illustrative but non-limiting example, the training may includeeducational content, such as work-related or career-related educationaltraining, as one example of educational content. An end user who isreceiving training may, for example, be wearing a VR or AR head display105, such as a VR headset or an AR headset. In the following discussion,it will be understood that both AR and VR implementations arecontemplated.

A set of biometric data is collected from the user to determinecognitive state metrics for the user related to learning efficacy. Anindividual's ability to learn effectively can be influenced by bothexternal factors and internal factors that have the potential tofacilitate or inhibit the learning process. External factors include theenvironment of the individual in the process of learning that could bedistracting or calming to the individual. Some examples of internalfactors include the mood of the individual, state of mind, energy level,or health of the individual. An individual's ability to learn alsovaries over time, and for some individuals the ability to learn can bedependent on the time of day. For example, some individuals might bemore effective learners early in the morning and less effective at latertimes in a day. Conversely, others might be more effective at latertimes of a day and less effective earlier in a day.

In one embodiment, the biometric data includes a measurement of theuser's brainwaves using an electroencephalogram (EEG) headset. Otherexamples of biometric data include eye tracking data, heart ratemeasurements, respiration, motion tracking, voice analysis, postureanalysis, facial analysis, and galvanic skin response. As illustrated inFIG. 1 , the environment about the user may include one or more sensorsto collect biometric data, such as one or more camera, microphones,heart monitors, etc. Some of the sensors may, for example, be built intothe AR/VR headset, as eye-tracking sensors, microphones, etc. Otherbiometric sensors may be worn on a user. Still other sensor may bedisposed in the general environment about the user, such as additionalcameras or microphones. Individual sensors may, for example, transmitsensor data via a wired channel, wireless channel, network interface,etc.

It should be noted that the raw biometric data typically doesn'tdirectly provide feedback on learning efficacy. Further processing ofeach source of biometric data is desirable to convert the biometric datainto a signal having attributes that are correlated or associated with alearning metric. As one example, the raw biometric data is ideallyanalyzed with respect to baseline data for the user. In the case of rawEEG data, the raw EEG data has different frequency bands (e.g., alpha,beta, and theta wave bands) and also spatial components in the sensethat a pattern of sensors over a user's head will have different signalcomponents from each sensor. This spatial pattern may, for example, besymmetric or asymmetric. Certain frequency bands may be generated morein certain parts of the brain than others. The spectral densities foreach channel may be determined and noise reduction techniques used.Ratios of the different bands may be determined, along with ratios orother aspects of the spatial dependencies. This can be used to determinea metric related to learning, such as a cognitive load indicative of howhard a user in thinking in a learning session. That is, the rawbiometric data is processed to generate a metric correlated to learningefficacy. Further processing permits the signals to be transformed intoa cognitive mental state metric for a particular attribute. For example,the metric could be a normalized number, such as a number within aselected range, such as a number between 0 and 1. The set of cognitivemental state metrics can thus be used to determine how to adapt a VR/ARtraining session.

As an example, raw EEG data may be processed to generate a metriccorresponding to a cognitive load, indicative of how “hard” the user isthinking based on the EEG data. However, a completer and more accuratepicture of the overall cognitive mental state of the user is generatedby including two or more different cognitive mental state metrics. Oneor more of these many be generated from other sources of non-EEGbiometric data. For example, heart rate, respiration, voice analysis,and galvanic skin response may be useful to generate cognitive mentalstate metrics associated with anxiety.

Some or all of the processing of the biometric data may be performed ina biometric analyzer 110. In one embodiment, the biometric analyzer 110has a processor, a memory, and computer software program code, stored onthe memory and executed by the processor, to implement classifiers toclassify the biometric data into the cognitive mental state metrics.This may be done separately for each source of biometric data. However,more generally, metrics may be generated from combinations of biometricdata inputs, such as using pre-programmed tables, matrices, or othertechniques to convert a combination of inputs from different datasources into at least one output indicative of a learning efficacy. Acommunication interface may be included in the biometric analyzer 110 toreceive biometric data from one or more biometric sensors. For example,a network communication interface may be provided to receive biometricdata from individual biometric sensors.

In some embodiments, a trained machine learning model may be used toperform the classification. Examples of a trained learning model aredescribed below in greater detail.

In one embodiment, a server 115 serves the content for the AR/VRtraining session. The server 115 may include computer processors, amemory, internal communication buses, and external communicationinterfaces. In some embodiments, the server 115 executes a neuroadaptiveVR learning program in which variations of an educational trainingsession (or set of sessions) are supported for a VR or AR educationaltraining session. The variations may, for example, including varying theeducational content type, content style, content complexity, pacing, orother factors of the AR/VR training. As one example a database 120 maybe provided to support variations in the training given based onbiometric data. For example, the database 120 could store two or moredifferent levels of complexity for one or more training modules of atraining session. More generally, the database 120 could store a matrixof different variations in training modules. Thus, in one embodiment,during a learning session, the VR learning program could accessdifferent pieces of content as a form of adaptation. In someembodiments, the database also stores, or aggregates, information on anindividual user's previous use of the system and for uses by otherparticipants. This historical data may, for example, be used to generatetraining data, as described below in more detail.

In one embodiment, a predictive engine 112 is included either in thebiometric analyzer 110 or server 115 to analyze the cognitive mentalstate metrics and determine when the training should be adapted. In thatsense, it functions as part of a training mode adapter to generatingtraining mode adaption commands. As examples, the predictive engine may,for example, include rules, tables, or a trained machine learning modelto examine a current set of cognitive mental state metrics anddetermined adjustments to the training session to maintain learningefficacy. In one embodiment, the predictive engine 112 is makingpredictions about the user's response to an educational session anddetermining training mode adaptations that may be required to maintainthe education session within an educational training zone having metricscompatible with effective learning.

As an illustrative example, suppose a cognitive load metric is risingand that an anxiety level metric is also rising. To prevent the userfrom become over-stressed, the predictive engine 112 proactively adaptsthe training session to maintain the cognitive mental state metricswithin a desired range associated with healthy learning. There are manydifferent ways this can be done. As one example a set of upper and lowerthresholds can be determined for individual cognitive mental statethresholds. However, more generally, dynamic aspects, such a rate ofchange, can be considered. The overall pattern of changes to a set ofcognitive mental state metrics can be considered. For example, a usermay currently be in a peak-learning mode but a rise in one or more ofthe cognitive mental state metrics may have trends that suggest that theuser's performance will degrade in the near future. In this situation,reducing the complexity of the education session at a point in timebefore the peak-learning mode ends may be a useful strategy.

For example, stress hormones can build up in the human body and take awhile to dissipate. Some individuals have a greater stress response thanothers. Moreover, a small fraction of the population is susceptible tobeing overwhelmed by stress in situations that trigger memories oftraumatic experience, such as Post Traumatic Stress Disorder (PTSD).Preventing a “spike” in anxiety/stress may be useful to maintain theoverall learning efficiency over the entire learning session. Thepredictive engine 112 could, for example, have upper and lower thresholdvalues selected with a margin below the absolute minimum and maximumvalues to provide a cushion for dealing with human response time. Forexample, suppose an “ideal” learning situation would be to maintain acognitive load below 90% and an anxiety level below 70%. However, inpractice, lower maximum thresholds might be chosen, such as maintaininga cognitive load below 80% and an anxiety level below 60% to reduce thepossibility of spiking behavior. In any case, the predictive engine 112may be implemented in different ways to have an algorithm that monitorsthe different cognitive mental state metrics and that proactively adaptsthe education session in response. This may be called “learning in theloop,” in the sense that the complexity of the educational content canbe dynamically adapted, based on the biometric data, before the learningexperience is substantially degraded. While a complexity level is oneexample of an adaptation, more generally other types of adaptation couldalso be performed. For example, a stress reduction technique could beinserted into the session, such as taking a break, playing a game, ordoing a breathing exercise.

FIG. 2 illustrates an example of a general predictive engine 112 in anembodiment in which the predictive engine 112 is part of a module thatgenerates training adaptation commands 111 based on an arbitrary numberof different cognitive mental state metrics and FIG. 3 illustrates anexample in which the cognitive mental state metrics include metricsgenerated by classifiers for a cognitive mental load, a motivationlevel, an anxiety level, and a focus level. It is an implementationdetail regarding the form of the training adaptation commands output bythe predictive engine 112. For example, the training adaptation commands111 could be a command in the form one or more numbers or codesindicating a desired mode change, such a complexity level (e.g., aninteger 1, 2, or 3 for high, medium, or low complexity as one example)to indicate changing to a different complexity mode. More generally, theoutput could be a set of normalized numbers, which are then used byother entities in the system to select training modules that areexecuted. For example, the predictive engine 112 could be implemented tooutput a complexity command code to signal an increase or decrease inthe complexity of the training. A rest break or relaxation command codeor number could be output to single the desirability of break in thetraining as another example. Other commands codes could also begenerated to account for common training scenarios. Alternatively, thepredictive engine could issue training adaptation commands in the formof direct decisions on particular training modules that are to be usedin a training session.

In one embodiment the predictive engine 112 is implemented as part of acontroller 114 that includes the predictive engine 112 and a predictivemodel 113 to select a training mode and generate the training adaptationcommand 111. For example, the controller 114 may include a processor,memory, and computer program instructions. In one embodiment, thepredictive model is a trained machine learning model, although moregenerally in may comprises table, matrices, or other features selectedto aid in making predictions about how to adapt an educational trainingsession.

As previously discussed, the predictive engine 112 may be implemented indifferent ways. And as described below in more detail, data from atraining session with the user (e.g., from a previous training session)may be used alone or in combination with data from other users as an aidto determine rules for making predictions. For example, in an enterprisetraining environment, a data set of a large number of participants maybe used to identify relationships between the monitored cognitive mentalstate metrics and learning efficacy.

Additionally, an individual user may be given an initial training testsession to obtain test data regarding their individual responses todifferent levels of test situations. For example, the training data(from a group of previous users) and the current user may be highlyspecific in terms of workforce demographics (e.g., blue collartechnicians), training objective (e.g., training of empathy in a jobinterfacing with the public under different situations). Having testdata and other data for the user and for a set of previous usersprovides a data set that can be used in different was to aid in makingpredictions.

In one embodiment, other non-biometric data may also optionally beutilized to aid in forming predictions, such as an aid in determiningtraining data. For example, some enterprises perform psychologicalassessment studies of employees using common tests such as the MyersBriggs test. For example, introverts may suffer more from anxiety in atraining for empathy than extroverts. Conversely, some extroverts maysuffer more anxiety doing multitasking in a technical environment.Regardless of whether other forms of data are used, the training/testdata generated for previous users and the current user can be selectedto provide data from which a prediction engine 112 adapts a trainingsession.

In one embodiment, a behavioral dashboard 125 provides a summary of thetraining and the cognitive mental state attributes/metrics during thetraining.

It will be understood that many possible implementations of thearchitecture of FIGS. 1, 2, and 3 are contemplated. For example, in anenterprise application, the server 115 and biometric analyzer 110 may beimplemented by the enterprise using enterprise client/serverarchitecture. However, one or both of the biometric analyze 110 andcontent server 115 could be implemented as a network service orcloud-based service. Moreover, as computing power increases, one or moreof the biometric analysis 110 and content server 115 may be implementedat least in part on local computers (e.g., a user's laptop computer,tablet computer, etc.) or at least in part on a headset (which may, forexample, include the headset used for the EEG).

As one example, a neuroadaptive VR learning program 117 supports amatrix of possible training variations. As one possibility, thecomplexity or pacing of the training could have two or more variations.However, more elaborate variations could be included to support a matrixof possibilities. For example, if a user becomes stressed in an empathyportion of a training session, a less challenging form of the trainingcould be performed. Alternatively, the sequence of training could bealtered to return to the remaining empathy training in a later portionof the training.

FIG. 4 is a flowchart of a general method in accordance with anembodiment. In block 405, the user signs in. In block 410 the biometricdata of the user is monitored. In block 415, a training phase isinitiated. One aspect of the training phase is that baseline data can beobtained to understand the user's response. This may include, forexample, performing one or more surveys or tests to assess a user'sbiometric responses. For example, the tests may provide generalpsychometric data regarding how the user responds to differentsituations. Additionally, the tests may be selected to be similar toaspects of the training. Individual testing is useful because of thevariations in individual behavior. For example, one individual may havehigh levels of background anxiety and stress in their life. Anotherindividual may have low levels of background anxiety and stress in theirlife. There are a variety of other reasons why individuals may responddifferently to a learning environment, including demographic factorslike age, level of education, previous experience with VR training, etc.

In block 420, an adaptive educational simulation is generated that isresponse to biometric data feedback. Thus, the cognitive mental statemetrics are maintained within a range conducive to learning.

FIG. 5 is a flow chart of a general method of using thresholds in thecognitive state metric(s) to make decisions to adapt the training. Inblock 505, a training phase is initiated to calibrate thresholds ofcognitive mental state metrics. The training phase may, for example,identify a threshold anxiety level predictive of a desirability toeither reduce or increase the complexity level or the pacing of thetraining. For example, if the anxiety level is below a first threshold,it may be predictive that the complexity of the training can beincreased, Conversely, if the anxiety level is above a second threshold,it may be predictive that the complexity of the training should bereduced to maintain an effective learning experience for the user.

In block 510, the virtual reality education phase is initiated. (Itwould be understood that in an alternate embodiment, the education phasemay use augmented reality). In block 515, the biometric data of the useris monitored during the education phase to generate feedback in the formof the cognitive state metrics. This monitoring may be performed on aperiodic basis at a rate that is fast compared with humanbehavioral/cognitive responses to educational training, e.g., once persecond.

The biometric data is used to provide feedback during the educationalsession to adapt the training. As previously discussed, this may includeadaptation such as selecting less complex training session modules,implementing relaxation breaks, changing in pacing or sequence, etc. Indecision block 520, the process ends when the training is completed.Otherwise, decisions are made whether the cognitive mental statethresholds are outside of upper or lower thresholds. If they are outsideof the thresholds, a decision is made in block 527 to adapt theeducation phase. Otherwise in block 530, the education phase iscontinued without adapting it.

FIG. 6 is a flowchart of another method of using thresholds to adapt theeducation phase. In block 605, thresholds are determined for at leastone cognitive mental state metric based on a cross-section ofparticipants, such as a set of participants who previously performedtesting or training exercises. This provides some information on alikely range of values for the cognitive mental state metrics to providean optimal zone for learning based on objectives such as efficacy oflearning, although other performance metrics could include, for example,enjoyment of learning.

In block 610, individual user baseline data and thresholds aredetermined during a test phase. This may also correspond to maintainingthe metric within a desired zone.

In block 615, an optional weighting may be performed using the thresholdinformation determined for the user and for the group of participants.

That is, the example of FIG. 6 would be compatible with using theindividual user data, data for a group of a previous users, or a hybridapproach in which the user's individual data is weighted with the datafor a group of participants.

One aspect of using a weighting approach is that it permits a reductionin the amount of training test phase needed for an individual. Forexample, a group of participants will have a distribution of responses.An individual user is likely to have responses falling (statistically)within the ranges of a statistically significant number of priorparticipants. Selecting a weighted value has some advantages over usingonly data from prior participants or only test phase data from the user.

In block 620, the weighted threshold upper and lower bounds are used todefine an optimal zone for learning corresponding to a set of learningobjectives.

Note that as an individual user continues to use the system, moreindividual data become available for them. For example, if a user goesthrough a series of lessons, there may be more historical data availablefrom which to determine upper and lower bound thresholds for the user.However, for a new user, there is less data available, and there may betime constraints on the amount of time a training test phase can beperformed. Thus, there may be use cases for which the weighted valuesprovide superior results.

As previously discussed, in one embodiment machine learning techniquesare utilized. FIG. 7 is flowchart of a machine learning method inaccordance with an embodiment. In block 705, training data is generatedfrom data collected for a set of prior users. For example, biometricdata from a set of users may be monitored during testing or trainingassociated with VR/AR training. For example, for a sales job, trainingdata could be collected for instances of a training program conducted ona set of prior users of the training program. In block 710, a machinelearning model is trained to classify biometric signals into signalsindicative of cognitive mental state metrics relevant to learning. Forexample, human users could label training data based on a combination ofobjective and subjective factors for learning as a whole (e.g., highlyeffective learning, effective learning, non-optimal learning,ineffective learning) or for specific cognitive mental state metrics(e.g., cognitive load, anxiety, etc.). For example, a human user couldlabel a section of the training, such as making an assessment if thestudent seemed over-anxious or was enjoying the training and learningwell. In block 715, the trained machine learning model is used toclassify biometric signals into signals indicative of a learning metric.Additional training could be performed regarding human users labelingsegments of data as points in time when a proactive change in thecomplexity of an education session would have been useful. That is, themachine learning model could be trained to predict when and how to adaptan education session.

FIG. 8 is a flowchart illustrating an example of transforming a rawbiometric signal for EEG data into a cognitive mental state metric.Block 805 includes pre-processing of the raw biometric data, which mayinclude aligning data, removing bad data, normalization, andinterpolation. In block 810, individual channels are selected. Forexample, with EEG data there may be different frequency band componentsthat correspond to different channels. Additionally, there may be aspatial response over different sensors on a user' head. Other steps810, 815, 820, 825, 830, 835, 840, 845, and 855 can be performed toanalyze the selected signal components in a spatio-spectraldecomposition, perform noise reduction and normalization step togenerate a cognitive load score. In this example, the selection ofchannels and other factors are chosen based on the cognitive statemetric that is being scored

In this example, it will be understood that empirical or heuristictechniques may be used to determine the associations between brainwavepatterns and cognitive state metrics relevant to learning. For example,aspects of brainwaves associated with a cognitive load may be based, inpart, on studying a group of test subjects who are objectively andsubjectively challenged by a learning exercise.

For example, EEG data may include different frequency componentscorresponding to different mental states. For example, differentfrequencies and patterns of brainwaves have different association withoverall mental activity and mental states. However, many other aspectsof brainwaves have correlations to different mental states that may berelevant to learning.

The efficacy of learning depends on many different factors, as indicatedby FIGS. 9-10 . For example, for many tasks, there are optimal ranges ofarousal. Too little arousal corresponds to fatigue, boredom orsleepiness. Another way to consider this is that there is an optimallevel of task difficulty, with respect to user capability, in order foroptimal learning performance to be achieved with low levels of anxietyand boredom.

Note that an individual educational training session may have aperformance metric defined for it. For example, maximization of learningcould be one objective. Alternatively achieving a balance abouteffective learning and joyful learning could be a different objective.For example, in a ten-part set of educational training sessions, itmight be a useful strategy to have a performance objective of joyfullearning in the first session to create a positive experience for user'snew to VR/AR training. Later sessions might emphasize maximal learning,as one example. There might even be individual examples in whichcreating high levels of anxiety or stress may be useful, such as in anempathy training course that has the objective of teaching what it islike to be anxious or over-stressed.

The use of the biometric data to provide feedback permits the trainingto dynamically adapt to optimize learning.

For example, certain quantitative sensor readings of biologicalfunctions, such as brain wave activity, heart rate and galvanic skinresponse, can be currently used to know if an individual is experiencingcertain qualitative dispositions, such as boredom or fatigue, emotionalengagement, motivation and attention.

However, there are practical difficulties in using brainwaves alone tocontrol. For example, eye tracking data is strongly correlated with userattention. Voice stress analysis can be used, in combination with otherbiometric data, to monitor stress/anxiety. For example, in an empathytraining for a customer service job, monitoring different types ofbiometric data permits broader insights than relying on a single type ofbiometric data. Using two or more different types of biometric datapermits a much richer understanding of cognitive mental states relatedto learning.

Learning is challenging because educational and training systems arebuilt for a mass or general audience, even though the way people mostsuccessfully learn varies from person to person, and from moment tomoment for any one individual. Additionally, the influx of technologicaldevices and constant communication into everyday lifestyles has led toincreased learning difficulties that may be related to attention,engagement, or sensory processing abilities.

There is longstanding research to support the idea that many individualsstruggle to succeed with learning or training curriculum designed for amass audience. There is also evidence of the detrimental impact thattechnology has had on our ability to retain and learn information.

The reactions of an individual to external and internal factors can bedetected by sensing biological feedback, such as brain waves, togenerate biological data to determine the state of an individual anddetermine how that state impacts the performance during a learningexercise.

Additional examples are described below for embodiments of different EEGheadset implementations.

1. Example 1: Adaptive Intelligent Learning System and Method UtilizingBiofeedback

An embodiment of the disclosure creates a new learning system including:individual data-gathering devices and methods; an aggregate, anonymizeddatabase of user performance; predictive models generated using theaggregate database that allow the system to synthesize and translateindividual data inputs into corresponding recommended actions; userdevices; and learning programs, all such components being connected viaa network or alternatively, not on a network depending on the desiredapplication. The learning system of an embodiment of the disclosureanalyzes and synthesizes user inputs that are gathered from sensors,surveys, observations and other sources to preferably augment, modify,or create a learning environment that is tailored to be most effectivefor the user at that specific moment in time.

Each time biological data is gathered from an individual completing alearning routine, that data becomes part of an aggregate database.Modeling that data using machine learning can create a system to predicthow a user will perform in a certain learning routine. This systemallows for scenarios where the individual data set that is beinggenerated in real time is compared against known trends in the aggregatedatabase in order to enact changes to the learning environment that arepredicted to improve that user's performance of the learning routine,such as changes to content style, color or speed. The system also givescommands to the user such as to take a break.

In an example of the learning system of an embodiment of the disclosure,the determination of effective learning conditions for a user isaccomplished by predicting a user's performance, based on a comparisonagainst known performance trends identified by the aggregate database.The predictive models that are created and continually refined using theaggregate database allow the system to create a real-time feedback loopthat drives learning programs to be modified in the course of anoperational routine. For example, consider students taking a readingcomprehension test. While taking the test, they are wearing a headsetthat collects brain wave data and feeds it into the database. Byanalyzing brain wave data, the system can identify if their workloadlevels spike into a territory that indicates cognitive load, which werecognize as overload or stress, or boredom. By comparing that indicatorto the database, the system can instantaneously predict what changeshould be enacted to reduce the individual user's cognitive load andsend a signal back to the learning software to enact a real-time changesuch as providing a simplified version of the same content prior to theuser completing a quiz of their comprehension of the section.

The learning system can also be used even when the environment isstatic, by analyzing user data such as brain waves to providesupplemental training activities or recommend next steps to improvelearning performance. In the reading comprehension example above, ratherthan enacting real-time changes to the system the user data could beanalyzed in comparison to the aggregate database to providerecommendations for follow up exercises to improve retention of thecontent. Further, over time the aggregate database can be used to createlearner profiles, and by matching individual users to those profilesthere are additional opportunities for tailored learning exercises basedon what the system knows about that profile's typical performance.

An example use case involves the gathering of qualitative andquantitative user input data before, during, or after a learning ortraining routine. The system is devised that it can take the processedinformation from a computational resource and detect user attributes,such as concentration, fatigue, stress, etc. and modifies an existing orcustom learning routine in order for the user to obtain improvedlearning outcomes such as better retention or performance.

In an embodiment, a headset is used to capture user data and thentransmit that data using an on-board computer to a networked server thathosts the aggregate database. This allows the system to conduct most ofthe heavy computing work in the cloud instead of on the headset, as wellas run predictive algorithms based on the aggregate database with acomputing power great enough to deliver a real-time signal back to theuser's device, which commands a change in the learning program in orderto make that program more effective.

The data that is gathered in real time is fit into a model and theresult of that model's output is sent back to the user. This could be inthe form of a visual, auditory, physical, or other type of indication,such as a change in content color, a sound indicating it is time to takea break, or changing content from text to drawings for visual learners.The output can also predict performance for the given task. Forinstance, in a reading comprehension example, the output could determinewhich areas in the assigned literature were comprehended more than otherareas, using timestamped data, image tracking, sensor readings, userinputs or the like to identify those areas. This would be accomplishedthrough factors being determined through the data such as concentration,energy level, and fatigue and then those factors being applied to themodel. This model is iterated over time in a continuous ornon-continuous fashion. This could be a background or foregroundfunction of the software.

Over time, each instance of making performance predictions based on userinputs and generating associated outputs will improve the system'sability to enhance individuals' performance within a given learningroutine. Users can input their own data sets to augment the machinelearning functions, such as surveys, independent sensor input, and othertext entries. Once learning challenges have been identified, thepredictive models could be applied to generate an associated feedback inthe learning program that is most strongly correlated with success. Thiscould be done with prompts, instructions, automated procedures, or thelike. For example, in one case, if inattention is identified in theindividual, they could be prompted with a simple instruction to stop andtake a 15-minute break before continuing with the learning or trainingexercise.

The type of data being gathered include examples such as brain wave datagathered using an EEG headset, self-reported assessments of learningeffectiveness and performance indicating mastery of the content taughtduring learning routines. These inputs are analyzed and then translatedinto a corresponding output action. In addition, using predictiveanalytics with behavioral triggers to reduce the compute power needed onthat device.

The predictive capability created by fitting the aggregated input datafrom each individual user into models using machine learning techniquesare unique because they identify learning challenges and predictappropriate output responses. These responses are delivered via thedevices that interface with the user to create a dynamic feedback loopthat improves learning outcomes.

In an embodiment of learning system and method of the presentdisclosure, a primary approach to acquire the necessary user informationor user input comes in the form of a device or a plurality of devicesplaced on the user's head in order to acquire the electrical activity inthe brain. One example of such a device could be an electroencephalogram(EEG).

As shown in FIGS. 11 a and 11 b , such devices can be assembled into anintegrated headset worn during use or, alternatively, be placed directlyonto the head without a headset. Additional sources of user inputs caninclude data gathered from keyboards, touch screens, verbal input, eyetracking, sensor data, or from external observation equipment such asscanners (optical or other) or user questionnaires/surveys.

Referring to FIGS. 12-17 , in an embodiment, a headset, in this example,is comprised of sensors, electrical and data acquisition/handlingsystem, and power storage electrically coupled together by wires on aheadband. In one embodiment, the headset includes powerhandling/charging, active cooling devices, and cooling inlets/outlets.

FIGS. 13 a and 13 b show schematic illustrations of the internalcomponents of an embodiment of a headset of the present invention havingan on-headset alerting device. This alerting device is intended tocommunicate with other devices, such as a cloud server for runningpredictive analytics programs, or directly with the user, or to holdinformation.

FIGS. 17 a and 17 b are schematic illustrations of the internalcomponents of an embodiment of a headset of the present invention withsensor blocks attached to the headset in a floating manner.

The sensors are preferably placed in understood point locations aroundthe scalp of a human head and also could include the ears as illustratedin FIGS. 18 and 19 . Sensors preferably do not require conductivepastes, liquids, or other medium to collect data. However, it isappreciated that those type of sensors can be used in the system tocollect the desired data.

As shown in FIG. 27 , a dry sensor can be surrounded by a soft material,such as for example, silicone, fabric, plastic, cotton, other paddingmaterial, or rubber or the sensor itself could be made entirely from asoft or flexible material. The sensor can be embedded in material or canbe attached to material. The sensor can be partially exposed, such as atthe tips, or be entirely covered by material or other covering.

In one embodiment, a headset itself enables autonomous calibration sothat a routine does not need to be run on the user in order to utilizethe data obtained from the user. This can be done through normalization,transforms of the data, or trend algorithms and is backed by the machinelearning routine that is created and updated over time. The signal gainsfrom the sensors are adjustable in order to have adequate data qualityto be utilized or sent to another device to be utilized. The signal canbe utilized without and with filtering such as physical or digitalfilters. Physical filters could come as a piece of electrical hardwareand digital filters could include software, a common example would be aKalman filter but could be generic or highly customized.

Referring to FIGS. 20-23 , in one embodiment, a user will complete alearning routine, either using a computer or another offline learningmethod, while wearing a headset device that may transmit data to otherdevices, or to the user, either while wired or wirelessly.

FIG. 20 is an illustrative example of a user wearing a headsetinterfacing with a computer wherein the headset feeds back into thecomputer wirelessly. This could also include routines in which data issent to a cloud server where it is processed and analyzed using machinelearning algorithms before sending some modification instruction back tothe learning software the user is studying. This data is processed offthe headset so that the overall compute power requirements are loweredand allow for a real-time change to be delivered to the user.

FIG. 21 is an illustrative example of a user wearing a headsetinterfacing with a computer wherein the headset feeds back into thecomputer over a wire. This could allow for other routines that arelocally processed in order to deliver feedback to the user.

FIG. 22 is an illustrative example of a user wearing a headsetinterfacing with a computer wherein the headset gives direct feedback toa user. For example, the colors used in a learning program could bechanged to highlight different sections of content.

FIG. 23 is an illustrative example of a user wearing a headsetinterfacing with non-connected material such as a book, wherein theheadset data feeds back into a computer wirelessly. This data could beused to provide recommendations to the user after they have finished thelearning routine, and will be added to the aggregate database to improvethe overall accuracy of functional models that are used in the machinelearning routines.

The software routine that is run could take many forms. The routinecould be run on the same device gathering inputs, a separate device, ora device running other software needed. One conceived permutationinvolves using the described innovation in order to enhance or augmentthe learning process. For example, the routine could involve a highlyefficient deep neural network model that can predict outcomes based oninputs. This routine can become more accurate over time through machinelearning operations that utilize the inputs to become more accurate inthe predictions as more data is gathered. The routine can also bestagnant and not change over time. For example, an enterprise may wantto train employees safe procedures for working in a facility. The systemmay be able to identify that based upon the speed with which a usercompletes the questions, they are not actually spending enough time toabsorb the information. The system may then be prompted to provide moredifficult testing questions that require critical reasoning, rathersimple questions that would only require a rote memorization of thecontent being presented.

The routines could utilize standard or quantum-based computing. Data forthe machine learning operations can be gathered from sensors like thedescribed headset or be given directly through a file upload or somesimilar operation.

For example, a server may include a processor(s) and a memory. Theprocessor(s) may be any suitable processing device, such as amicroprocessor, microcontroller, integrated circuit, or other suitableprocessing device. Similarly, the memory may include any suitablecomputer-readable medium or media, including, but not limited to,non-transitory computer-readable media, RAM, ROM, hard drives, flashdrives, or other memory devices. The memory may store informationaccessible by processor(s), including instructions that can be executedby processor(s) and data that can be retrieved, manipulated, created, orstored by processor(s). In several embodiments, the data may be storedin one or more databases.

Similar to the server, the user device may also include one or moreprocessors and associated memory. The processor(s) may be any suitableprocessing device known in the art, such as a microprocessor,microcontroller, integrated circuit, or other suitable processingdevice. Similarly, the memory may be any suitable computer-readablemedium or media, including, but not limited to, non-transitorycomputer-readable media, RAM, ROM, hard drives, flash drives, or othermemory devices. As is generally understood, the memory may be configuredto store various types of information, such as data that may be accessedby the processor(s) and instructions that may be executed by theprocessor(s). The data may generally correspond to any suitable files orother data that may be retrieved, manipulated, created, or stored byprocessor(s). In several embodiments, the data may be stored in one ormore databases.

An embodiment of the present disclosure also provides for an on-devicecomputer that is capturing data and sending that data to another serverwhere the aggregate database lives. The heavy computing work ofprocessing and analyzing that data, using machine learning to createparameters to fit the predictive model, and then sending a signal can behandled by a server. Predictive analytics with behavioral triggersreduce the compute power required on a device.

FIG. 24 is a flowchart of an embodiment of a machine learning routine ofthe present disclosure. In the machine learning routine, dynamic userinput data such as brain wave activity, galvanic skin response, heartrate and other biological sources, is fit to preset boundary conditionsset based on static outside data such as the aggregate database. Theseinputs are fed into the machine learning routine to pre-process thedata, predict which response would help improve the user's performance,then deliver a solution back to the user. At the end of the routine, thesystem will evaluate the user success of the routine and log thatperformance data to improve future performance of the predictivealgorithm.

FIG. 25 is a flowchart of an embodiment of a functional model operationof the present disclosure. In the functional model operation dynamicuser input data is fed into the model, and a predicted outcome isgenerated based on historical data and boundary conditions are createdusing the machine learning routine. The output is continually fed backinto the model.

FIG. 26 is a flowchart of an embodiment of a possible lesson in the loopwith the user in accordance with the present disclosure. Data iscollected from the user while they execute the learning routine, andanalyzed to determine attributes such as cognitive load or emotionalengagement. Depending on how the user data performs relative to presetboundary conditions, the lesson material may be modified in some way toimprove performance, such as by delivering an alert to the user to stopand take a break.

2. Example 2: Network Connecting People and Headset-Based Brain-ComputerInterface

An embodiment of the disclosure is directed to a learning system andmethod that is tailored to an individual user, or a group of users, toprovide an effective learning program. The learning system and method ofthe present invention comprises many possible formulations andcombinations of the following that can be used to solve the challengeswith standardized approaches to learning and training by helpingindividuals or groups enter or maintain a mental state that is predictedto have the best learning outcomes, such as improved retention or taskperformance. The system and method of the present disclosure provide anopportunity to gain even stronger insights about how people are learningin group settings, and how individual performance may be impacted in agroup setting, by connecting multiple headsets that act as an interfacebetween the user and the computer they are using to learn.

In one embodiment, the system and method of the present disclosurecreates a new learning system by connecting multiple individualdata-gathering devices and methods so that they can communicate betweenthemselves. In this setting, individual and group data will becontributed to an aggregate, anonymized database of user performance;predictive models generated using the aggregate database that allow thesystem to synthesize and translate individual or group data inputs intocorresponding recommended actions; user devices; and learning programs,all such components being connected via a network or, alternatively, noton a network depending on the desired application. The learning systemof the present invention analyzes and synthesizes user inputs that aregathered from sensors, surveys, observations and other sources topreferably augment, modify, or create a learning environment that istailored to be most effective for the user at that specific moment intime.

Each time biological data is gathered from an individual completing alearning routine, that data becomes part of an aggregate database.Modeling that data using machine learning can create a system to predicthow a user will perform in a certain learning routine. This system wouldallow for scenarios where the individual data set that is beinggenerated in real time is compared against known trends in the aggregatedatabase in order to enact changes to the learning environment that arepredicted to improve that user's performance of the learning routine,such as changes to content style, color or speed. The system also givescommands to the user such as to take a break.

In an embodiment of the learning system of the present disclosure, thedetermination of effective learning conditions for a user isaccomplished by predicting a user's performance along with a comparisonto their learning group, based on a comparison against known performancetrends identified by the aggregate database. The predictive models thatare created and continually refined using the aggregate database allowthe system to create a real-time feedback loop that drives learningprograms to be modified in the course of an operational routine. Forexample, consider students working together in a team learning scenario,where tasks and functions are spread throughout a group. While workingtogether, they are wearing a headset that collects brain wave data andfeeds it into the database. By analyzing brain wave data, the system canidentify if their workload levels spike into a territory that indicatescognitive load, which we recognize as overload or stress, or boredom. Bycomparing that indicator to the database, the system can instantaneouslypredict what change should be enacted to reduce the individual user'scognitive load and send a signal back to the learning software to enacta real-time change such as designating a member of the team to take aturn as leader, or providing a simplified version of the same contentprior to the individual user completing a quiz of their comprehension ofthe section.

In one embodiment, a primary use case involves the gathering ofqualitative and quantitative user input data before, during or after alearning or training routine. The system is devised so that it can takethe processed information from a computational resource and detect userattributes, such as concentration, fatigue, stress, etc. and modifies anexisting or custom learning routine in order for the user to obtainimproved learning outcomes such as better retention or performance.

In one embodiment, a headset is used to capture user data and thentransmit that data using an on-board computer to a networked server thathosts the aggregate database. Multiple individuals may wear headsets,such that the headsets are able to communicate between one another. Inaddition to facilitating team learning scenarios as described above, theinteraction of users wearing headsets could also be used to facilitateteam learning through competition, as well as predicting behavior likecheating.

The data that is gathered in real time is fit into a model and theresult of that model's output is sent back to the individual user and/orgroup. This could be in the form of a visual, auditory, physical, orother type of indication, such as a change in content color, a soundindicating it is time to take a break, or changing content from text todrawings for visual learners. The output can also predict performancefor the given task. For instance, in a reading comprehension example,the output could determine which areas in the assigned literature werecomprehended more than other areas, using timestamped data, imagetracking, sensor readings, user inputs or the like to identify thoseareas. This would be accomplished through factors being determinedthrough the data such as concentration, energy level, and fatigue andthen those factors being applied to the model. This model is iteratedover time in a continuous or non-continuous fashion. This could be abackground or foreground function of the software.

Over time, each instance of making performance predictions based on userinputs and generating associated outputs will improve the system'sability to enhance individuals' and group performance within a givenlearning routine. Users can input their own data sets to augment themachine learning functions, such as surveys, independent sensor input,and other text entries. Once learning challenges have been identified,the predictive models could be applied to generate an associatedfeedback in the learning program that is most strongly correlated withsuccess. This could be done with prompts, instructions, automatedprocedures, or the like. For example, in one case, if inattention isidentified in the individual, they could be prompted with a simpleinstruction to stop and take a 15-minute break before continuing withthe learning or training exercise. If one team member is identified tobe intimidating to other team members, the identified team member mayreceive additional training on effectively collaborating in a teamsetting.

The type of data being gathered include examples such as brain wave datagathered using an EEG headset, self-reported assessments of learningeffectiveness and performance indicating mastery of the content taughtduring learning routines. These inputs are analyzed and then translatedinto a corresponding output action. In addition, using predictiveanalytics with behavioral triggers to reduce the compute power needed onthat device.

The predictive capability created by fitting the aggregated input datafrom each individual user into models using machine learning techniquesare unique because they identify learning challenges and predictappropriate output responses. These responses are delivered via thedevices that interface with the user to create a dynamic feedback loopthat improves learning outcomes.

In a preferred embodiment of learning system and method of the presentinvention, a primary approach to acquire the necessary user informationor user inputs comes in the form of a device or a plurality of devicesplaced on the user's head in order to acquire the electrical activity inthe brain. One example of such a device could be an electroencephalogram(EEG).

As shown in FIGS. 11 a and 11 b , such devices can be assembled into anintegrated headset worn during use or, alternatively, be placed directlyonto the head without a headset. Additional sources of user inputs caninclude data gathered from keyboards, touch screens, verbal input, eyetracking, sensor data, or from external observation equipment such asscanners (optical or other) or user questionnaires/surveys.

Referring to FIGS. 12-17 , in one embodiment, a headset 100, in thisexample, is comprised of sensors, electrical and dataacquisition/handling system, and power storage electrically coupledtogether by wires on a headband. In one embodiment, the headset includespower handling/charging, active cooling devices, and coolinginlets/outlets.

FIGS. 13 a and 13 b show schematic illustrations of the internalcomponents of an embodiment of a headset of the present invention havingan on-headset alerting device. This alerting device is intended tocommunicate with other devices, such as a cloud server for runningpredictive analytics programs, or directly with the user, or to holdinformation.

FIGS. 17 a and 71 b are schematic illustrations of the internalcomponents of an embodiment of a headset of the present invention withsensor blocks attached to the headset in a floating manner.

The sensors are preferably placed in understood point locations aroundthe scalp of a human head and also could include the ears as illustratedin FIGS. 18 and 19 . The sensors preferably do not require conductivepastes, liquids, or other medium to collect data. However, it isappreciated that those type of sensors can be used in the system tocollect the desired data.

For example, a sensor could be a dry sensor. it can be surrounded by asoft material, such as for example, silicone, fabric, plastic, cotton,other padding material, or rubber, or sensor itself could be madeentirely from a soft or flexible material. The sensor can be embedded inmaterial or can be attached to material. The sensor can be partiallyexposed, such as at the tips, or be entirely covered by material orother covering.

The headset itself enables autonomous calibration so that a routine doesnot need to be run on the user in order to utilize the data obtainedfrom the user. This can be done through normalization, transforms of thedata, or trend algorithms and is backed by the machine learning routinethat is created and updated over time. The signal gains from the sensorsare adjustable in order to have adequate data quality to be utilized orsent to another device to be utilized. The signal can be utilizedwithout and with filtering such as physical or digital filters. Physicalfilters could come as a piece of electrical hardware and digital filterscould include software, a common example would be a Kalman filter butcould be generic or highly customized.

Referring to FIGS. 20-23 , in one embodiment, a user will complete alearning routine, either using a computer or another offline learningmethod, while wearing a headset device that may transmit data to otherdevices, or to the user, either while wired or wirelessly.

FIG. 20 is an illustrative example of a user wearing a headsetinterfacing with a computer wherein the headset feeds back into thecomputer wirelessly. This could also include routines in which data issent to a cloud server where it is processed and analyzed using machinelearning algorithms before sending some modification instruction back tothe learning software the user is studying. This data is processed offthe headset so that the overall compute power requirements are loweredand allow for a real-time change to be delivered to the user.

FIG. 21 is an illustrative example of a user wearing a headsetinterfacing with a computer wherein the headset feeds back into thecomputer over a wire. This could allow for other routines that arelocally processed in order to deliver feedback to the user.

FIG. 22 is an illustrative example of a user wearing a headsetinterfacing with a computer wherein the headset gives direct feedback toa user. For example, the colors used in a learning program could bechanged to highlight different sections of content.

FIG. 23 is an illustrative example of a user wearing a headsetinterfacing with non-connected material such as a book, wherein theheadset data feeds back into a computer wirelessly. This data could beused to provide recommendations to the user after they have finished thelearning routine, and will be added to the aggregate database to improvethe overall accuracy of functional models that are used in the machinelearning routines.

The software routine that is run could take many forms. The routinecould be run on the same device gathering inputs, a separate device, ora device running other software needed. One conceived permutationinvolves using the described innovation in order to enhance or augmentthe learning process. For example, the routine could involve a highlyefficient deep neural network model that can predict outcomes based oninputs. This routine can become more accurate over time through machinelearning operations that utilize the inputs to become more accurate inthe predictions as more data is gathered. The routine can also bestagnant and not change over time. For example, an enterprise may wantto train employees safe procedures for working in a facility. The systemmay be able to identify that based upon the speed with which a usercompletes the questions, they are not actually spending enough time toabsorb the information. The system may then be prompted to provide moredifficult testing questions that require critical reasoning, rathersimple questions that would only require a rote memorization of thecontent being presented.

The routines could utilize standard or quantum-based computing. Data forthe machine learning operations can be gathered from sensors like thedescribed headset or be given directly through a file upload or somesimilar operation.

For example, a server may include a processor(s) and a memory. Theprocessor(s) may be any suitable processing device, such as amicroprocessor, microcontroller, integrated circuit, or other suitableprocessing device. Similarly, the memory may include any suitablecomputer-readable medium or media, including, but not limited to,non-transitory computer-readable media, RAM, ROM, hard drives, flashdrives, or other memory devices. The memory may store informationaccessible by processor(s), including instructions that can be executedby processor(s) and data that can be retrieved, manipulated, created, orstored by processor(s). In several embodiments, the data may be storedin one or more databases.

Similar to the server, the user device may also include one or moreprocessors and associated memory. The processor(s) may be any suitableprocessing device known in the art, such as a microprocessor,microcontroller, integrated circuit, or other suitable processingdevice. Similarly, the memory may be any suitable computer-readablemedium or media, including, but not limited to, non-transitorycomputer-readable media, RAM, ROM, hard drives, flash drives, or othermemory devices. As is generally understood, the memory may be configuredto store various types of information, such as data that may be accessedby the processor(s) and instructions that may be executed by theprocessor(s). The data may generally correspond to any suitable files orother data that may be retrieved, manipulated, created, or stored byprocessor(s). In several embodiments, the data may be stored in one ormore databases.

An embodiment of the present invention also provides for an on-devicecomputer that is capturing data and sending that data to another serverwhere the aggregate database lives. The heavy computing work ofprocessing and analyzing that data, using machine learning to createparameters to fit the predictive model, and then sending a signal can behandled by a server. Predictive analytics with behavioral triggersreduce the compute power required on a device.

FIGS. 28 and 29 show embodiments of the headset of the present inventionthat can process data on the headset itself using an on-board computer,or conversely, by wirelessly transmitting that data to an externalserver for analysis and execution of the learning system scheme.

In an embodiment, the computer is on board the headset directly. Theon-board computer can be made to be removable or permanently attached.The computer will have storage and processing capabilities, as well aswireless communication to facilitate the heaviest computing work at anetworked location. Machine learning routines established at a networkedlocation can run on the computer for direct improvement of the learningenvironment (lesson) and can be used to interpret the user's uniquesignatures for a given stimulus (aide for self-calibration).

The computer can be detached and signals can go between the headset andthe computer, vice versa, or two-way communication between the two.

FIGS. 30 through 34 show examples of various possible scenarios to shareinformation among multiple headsets being worn by individual users, andwith the central server location where data is processed and machinelearning schemes executed. This arrangement would support team learningactivities as described, and will allow for the predictive algorithms tofunction during the course of a lesson between participants. Thecomputer can be on-board the headset or operate independently, andcommunicate with the lesson material to alter the experience in a waythat has been predicted to improve learning outcomes.

The headsets are able to communicate between themselves using on-boardcomputers with wireless communication abilities, as well as withnetworked computers. This can support a facilitated team-learningscenario where group members each have different and dependentresponsibilities. It can also be used to predict attitudes and behaviorssuch as cohesion, or cheating. Machine learning can be used to makepredictive recommendations to improve learning outcomes based on theindividual and group responses assessed during a group lesson. This maybe experienced by the student when they are using a computer that isnetworked with the headset, and a recommendation triggers a change inthe software they are using to access a learning program.

The addition of direct visual output can be added to the headsets toallow communicated information to be shared between users, such as alight or color to indicate stress or excitement.

Headsets or computers connected to headsets can transmit data back to acentral location for compiling or learning scheme execution. Thiscentral location will update the models and modify learning executionscenarios. The central location can be one or many networked locations.Similarly, the network can be disaggregated into the headsetsthemselves. Back and forth communication is optional for each scenariodescribed. The headsets do not have to be in a single location tobenefit from the collective data gathering. Conversely, headsets can belimited to a single location for more customized applications.

Predictive Model-Based System for Lower On-Device Compute Power

In one embodiment, the learning system and method comprises a computingsystem that includes a wearable device such as a headset with anintegrated on-board computer and wireless communication system, and apredictive model engine that can execute complex problems on a networkedcloud server, and a feedback loop to the user headset and learningdevice.

In one embodiment, the learning system and method may include gatheringuser data from the individual's headset, wirelessly transmitting thatdata to an external server for analysis, running predictive models basedon that data and delivering a response back to the headset in order totrigger a response. In the preferred embodiment, this directive would bepredicted to improve learning outcomes based on machine learningroutines built using the behavioral database and realized within thelearning environment. Processing this data off the headset allows theentire routine to be run in real-time for the user, while enablingminimum power requirements on the user headset.

It is appreciated that an embodiment could also be used for otherapplications such as enabling other wearable devices and sensors thatgather biological data to similarly conduct heavy computing work in thecloud, while minimizing the size and computing requirements for thewearable device.

The learning system and method may be tailored to an individual user, ora group of users, to provide an effective learning program. The learningsystem and method comprise many possible formulations and combinationsof the following that can be used to solve the challenges withstandardized approaches to learning and training by helping individualsor groups enter or maintain a mental state that is predicted to have thebest learning outcomes, such as improved retention or task performance.

For example, the learning system and method in one embodiment includes:individual data-gathering devices and methods; an aggregate, anonymizeddatabase of user performance; predictive models generated using theaggregate database that allow the system to synthesize and translateindividual data inputs into corresponding recommended actions; userdevices; and learning programs, all such components being connected viaa network or alternatively, not on a network depending on the desiredapplication. The predictive model-based system of the present inventionallows data-gathering devices such as an electroencephalogram (EEG) forexample, or other types of wearable devices to analyze and synthesizeuser inputs in real-time, while maintaining comfort and usability ofsuch devices. In the preferred embodiment, this real-time signal wouldbe used preferably to augment, modify, or create a learning environmentthat is predicted to be most effective for the user at that specificmoment in time.

Each time data is gathered from an individual using the learning system,in this example gathering biological data to complete a learningroutine, that data becomes part of an aggregate database, which becomesthe foundational dataset for the predictive algorithms. Modeling thatdata using machine learning can create a system to predict how a userwill perform in a certain learning routine. This predictive system wouldallow for scenarios where the individual data set that is beinggenerated in real time is compared against known trends in the aggregatedatabase in order to enact changes to the learning environment that arepredicted to improve that user's performance of the learning routine,such as changes to content style, color or speed. These complex problemsare best suited to processing by a remote processor such as an externalserver and off the on-board computer of a user device to allow forlightweight hardware that can still deliver high-powered predictiverecommendations.

In one embodiment of the learning system, the determination of effectivelearning conditions for a user is accomplished by predicting a user'sperformance, based on a comparison against known performance trendsidentified by the aggregate database. The predictive models that arecreated and continually refined using the aggregate database allow thesystem to create a real-time feedback loop that drives learning programsto be modified in the course of an operational routine. For example,consider students taking a reading comprehension test. While taking thetest, they are wearing a headset that collects brain wave data and feedsit into the database. By analyzing brain wave data, the system canidentify if their workload levels spike into a territory that indicatescognitive load, which we recognize as overload or stress, or boredom. Bycomparing that indicator to the database, the system can instantaneouslypredict what change should be enacted to reduce the individual user'scognitive load and send a signal back to the learning software to enacta real-time change such as providing a simplified version of the samecontent prior to the user completing a quiz of their comprehension ofthe section. Over time the aggregate database can be used to createlearner profiles, and by matching individual users to those profilesthere are additional opportunities for tailored learning exercises basedon what the system knows about that profile's typical performance.

In one embodiment, a headset is used to capture user data and thentransmit that data using an on-board computer to a networked server thathosts the aggregate database. This allows the system to conduct most ofthe heavy computing work in the cloud instead of on the headset, as wellas run predictive algorithms based on the aggregate database with acomputing power great enough to deliver a real-time signal back to theuser's device, which commands a change in the learning program in orderto make that program more effective.

The primary use case involves the gathering of qualitative andquantitative user input data before, during or after a learning ortraining routine. The system is devised that it can take the processedinformation from a computational resource, send that data to a networkedserver and run machine learning routines to predict likely user behaviorbased on user attributes, such as concentration, fatigue, stress, etc.

The data that is gathered in real time is fit into a model and theresult of that model's output is sent back to the user. When used aspart of the learning system, this signal to the user could be in theform of a visual, auditory, physical, or other type of indication, suchas a change in content color, a sound indicating it is time to take abreak, or changing content from text to drawings for visual learners.The output can also predict performance for the given task. Forinstance, in a reading comprehension example, the output could determinewhich areas in the assigned literature were comprehended more than otherareas, using timestamped data, image tracking, sensor readings, userinputs or the like to identify those areas. This would be accomplishedthrough factors being determined through the data such as concentration,energy level, and fatigue and then those factors being applied to themodel. This model is iterated over time in a continuous ornon-continuous fashion. This could be a background or foregroundfunction of the software.

Over time, each instance of making performance predictions based on userinputs and generating associated outputs will improve the system'sability to apply predictive models and generate an associated feedbackin the learning program that is most strongly correlated with success.This could be done with prompts, instructions, automated procedures, orthe like. For example, in one case, if inattention is identified in theindividual, they could be prompted with a simple instruction to stop andtake a 15-minute break before continuing with the learning or trainingexercise.

The type of data being gathered include examples such as brain wave datagathered using an EEG headset, or other biological data gathered usingwearable devices. These inputs are analyzed and then translated into acorresponding output action. Using predictive analytics based on thesebehavioral triggers will reduce the compute power needed on that device.

The predictive capability created by fitting the aggregated input datafrom each individual user into models using machine learning techniquesare unique because they identify learning challenges and predictappropriate output responses. These responses are delivered via thedevices that interface with the user to create a dynamic feedback loopthat improves learning outcomes.

These and other objects of the present invention will be apparent fromreview of the following specification and the accompanying drawings.

In a preferred embodiment of learning system and method of the presentinvention, a primary approach to acquire the necessary user informationor user inputs comes in the form of a device or a plurality of devicesplaced on the user's head in order to acquire the electrical activity inthe brain. One example of such a device could be an electroencephalogram(EEG).

As shown in FIGS. 11 a and 11 b , such devices can be assembled into anintegrated headset worn during use or, alternatively, be placed directlyonto the head without a headset. Additional sources of user inputs caninclude data gathered from keyboards, touch screens, verbal input, eyetracking, sensor data, or from external observation equipment such asscanners (optical or other) or user questionnaires/surveys.

Referring to FIGS. 12, 17, 18, 19, 20, 21 in one embodiment, a headset,in this example, is comprised of sensors, an electrical and dataacquisition/handling system, and power storage electrically coupledtogether by wires on a headband. In one embodiment, headset includespower handling/charging active cooling devices, and coolinginlets/outlets.

FIGS. 12 a and 12 b show schematic illustrations of the internalcomponents of an embodiment of headset of the present invention havingan on-headset alerting device. This alerting device is intended tocommunicate with other devices, such as a cloud server for runningpredictive analytics programs, or directly with the user, or to holdinformation.

FIGS. 17 a and 17 b are schematic illustrations of the internalcomponents of an embodiment of a headset of the present invention withsensor blocks attached to a headset in a floating manner.

The headset itself enables autonomous calibration so that a routine doesnot need to be run on the user in order to utilize the data obtainedfrom the user. This can be done through normalization, transforms of thedata, or trend algorithms and is backed by the machine learning routinethat is created and updated over time. The signal gains from the sensorsare adjustable in order to have adequate data quality to be utilized orsent to another device to be utilized. The signal can be utilizedwithout and with filtering such as physical or digital filters. Physicalfilters could come as a piece of electrical hardware and digital filterscould include software, a common example would be a Kalman filter butcould be generic or highly customized.

Referring to FIGS. 20, 21, 22, and 23 , in an embodiment, a user willcomplete a learning routine, either using a computer or another offlinelearning method, while wearing a headset device that may transmit datato other devices, or to the user, either while wired or wirelessly.

FIG. 20 is an illustrative example of a user wearing a headsetinterfacing with a computer wherein the headset feeds back into thecomputer wirelessly. This could also include routines in which data issent to a cloud server where it is processed and analyzed using machinelearning algorithms before sending some modification instruction back tothe learning software the user is studying. This data is processed offthe headset so that the overall compute power requirements are loweredand allow for a real-time change to be delivered to the user.

FIG. 21 is an illustrative example of a user wearing a headsetinterfacing with a computer wherein the headset feeds back into thecomputer over a wire. This could allow for other routines that arelocally processed in order to deliver feedback to the user.

FIG. 22 is an illustrative example of a user wearing a headsetinterfacing with a computer wherein the headset gives direct feedback toa user. For example, the colors used in a learning program could bechanged to highlight different sections of content.

FIG. 23 is an illustrative example of a user wearing a headsetinterfacing with non-connected material such as a book, wherein theheadset data feeds back into a computer wirelessly. This data could beused to provide recommendations to the user after they have finished thelearning routine, and will be added to the aggregate database to improvethe overall accuracy of functional models that are used in the machinelearning routines.

The software routine that is run could take many forms. The routinecould be run on the same device gathering inputs, a separate device, ora device running other software needed. One conceived permutationinvolves using the described innovation in order to enhance or augmentthe learning process. For example, the routine could involve a highlyefficient deep neural network model that can predict outcomes based oninputs. This routine can become more accurate over time through machinelearning operations that utilize the inputs to become more accurate inthe predictions as more data is gathered. The routine can also bestagnant and not change over time. For example, an enterprise may wantto train employees safe procedures for working in a facility. The systemmay be able to identify that based upon the speed with which a usercompletes the questions, they are not actually spending enough time toabsorb the information. The system may then be prompted to provide moredifficult testing questions that require critical reasoning, rather thansimple questions that would only require a rote memorization of thecontent being presented.

The routines could utilize standard or quantum-based computing. Data forthe machine learning operations can be gathered from sensors like thedescribed headset or be given directly through a file upload or somesimilar operation.

For example, a server may include a processor(s) and a memory. Theprocessor(s) may be any suitable processing device, such as amicroprocessor, microcontroller, integrated circuit, or other suitableprocessing device. Similarly, the memory may include any suitablecomputer-readable medium or media, including, but not limited to,non-transitory computer-readable media, RAM, ROM, hard drives, flashdrives, or other memory devices. The memory may store informationaccessible by processor(s), including instructions that can be executedby processor(s) and data that can be retrieved, manipulated, created, orstored by processor(s). In several embodiments, the data may be storedin one or more databases.

Similar to the server, the user device may also include one or moreprocessors and associated memory. The processor(s) may be any suitableprocessing device known in the art, such as a microprocessor,microcontroller, integrated circuit, or other suitable processingdevice. Similarly, the memory may be any suitable computer-readablemedium or media, including, but not limited to, non-transitorycomputer-readable media, RAM, ROM, hard drives, flash drives, or othermemory devices. As is generally understood, the memory may be configuredto store various types of information, such as data that may be accessedby the processor(s) and instructions that may be executed by theprocessor(s). The data may generally correspond to any suitable files orother data that may be retrieved, manipulated, created, or stored byprocessor(s). In several embodiments, the data may be stored in one ormore databases.

In one embodiment, the system also provides for an on-device computerthat is capturing data and sending that data to another server where theaggregate database lives. The heavy computing work of processing andanalyzing that data, using machine learning to create parameters to fitthe predictive model, and then sending a signal can be handled by aserver. Predictive analytics with behavioral triggers reduce the computepower required on a device.

FIG. 24 is a flowchart of an embodiment of a machine learning routine.In the machine learning routine, dynamic user input data such as brainwave activity, galvanic skin response, heart rate and other biologicalsources, is fit to preset boundary conditions set based on staticoutside data such as the aggregate database. These inputs are fed intothe machine learning routine to pre-process the data, predict whichresponse would help improve the user's performance, then deliver asolution back to the user. At the end of the routine, the system willevaluate the user success of the routine and log that performance datato improve future performance of the predictive algorithm.

FIG. 25 is a flowchart of an embodiment of a functional model operationof the present invention. In the functional model operation dynamic userinput data is fed into the model, and a predicted outcome is generatedbased on historical data and boundary conditions are created using themachine learning routine. The output is continually fed back into themodel.

FIG. 26 is a flowchart of an embodiment of a possible lesson in the loopwith the user in accordance with the present invention. Data iscollected from the user while they execute the learning routine, andanalyzed to determine attributes such as cognitive load or emotionalengagement. Depending on how the user data performs relative to presetboundary conditions, the lesson material may be modified in some way toimprove performance, such as by delivering an alert to the user to stopand take a break.

Some additional examples will now be described:

Example 1

A computer-implemented method comprising:

-   -   monitoring biometric data of a user during a virtual reality or        augmented reality educational training session, the biometric        data including electroencephalogram (EEG) data;    -   classifying the biometric data into at least one metric of at        least one cognitive mental state associated with a learning        efficacy; and    -   adapting the virtual reality or augmented reality training        session based on values of the at least one metric, with the        adapting being selected on a learning performance prediction        based on the at least one cognitive mental state metric.

Example 2

The computer implemented method of Example 1, wherein the biometric datafurther comprises at least one of a heart rate, eye tracking data, andmotion tracking data.

Example 3

The computer implemented method of Example 1, wherein the at least onemetric comprises a cognitive load.

Example 4

The computer implemented method of Example 1, wherein the at least onemetric further comprises at least one of an anxiety level, a motivationlevel, a focus level, and an attention level.

Example 5

The computer implemented method of Example 1, wherein the adapting istriggered based on at least one threshold value of the at least onemetric.

Example 6

The computer implemented method of Example 2, further comprisingcalibrating thresholds of the at least one metric for the user in a testphase.

Example 7

The computer implemented method of Example 1, wherein the test phasecomprises monitoring the biometric data during the test phase inresponse to training material presented via the virtual reality oraugmented reality.

Example 8

The computer implemented method of Example 1, wherein the adapting isselected to maintain the at least one cognitive mental state within aselected range of values.

Example 9

The computer implemented method of Example 1, further comprisingutilizing a trained machine learning model to perform the classifyingand the adapting.

Example 10

The computer implemented method of Example 1, wherein the adaptingcomprises at least one of adapting a complexity of the learning session,adapting a pace of the learning session, and incorporating rest periodsin the learning session.

Example 11

A system comprising:

-   -   a biometric data monitor to monitor biometric data of a user        during a virtual reality or augmented reality training session,        the biometric data including electroencephalogram (EEG) data;    -   a classifier to classify the biometric data into at least one        metric of at least one cognitive mental state associated with a        learning efficacy; and    -   a content server to serve content to a virtual reality or        augmented reality system, the content server adapting the        content of a training session based on values of the at least        one metric, with the adapting being selected on a learning        performance prediction based on the at least one cognitive        mental state metric.

Example 12

The system of Example 11, wherein the biometric data further comprisesat least one of a heart rate, eye tracking data, and motion trackingdata.

Example 13

The system of Example 11, wherein the at least one metric comprises acognitive load.

Example 14

The system of Example 11, wherein the at least one metric furthercomprises at least one of an anxiety level, a motivation level, a focuslevel, and an attention level.

Example 15

The system of Example 11, wherein the adapting is triggered based on atleast one threshold of the at least one metric.

Example 16

The system of Example 11, further comprising calibrating thresholds ofthe at least one metric for the user in a test phase.

Example 17

The system of Example 11, wherein the test phase comprises monitoringthe biometric data during the test phase in response to trainingmaterial presented via the virtual reality or augmented reality.

Example 18

The system of Example 11, wherein the adapting is selected to maintainthe at least one cognitive mental state within a selected range ofvalues.

Example 19

The system of Example 11, further comprising utilizing a trained machinelearning model to perform the classifying and the adapting.

Example 20

The system of Example 11, wherein the adapting comprises at least one ofadapting a complexity of the learning session, adapting a pace of thelearning session, and incorporating rest periods in the learning session

Example 21

A computer-implemented method comprising:

-   -   storing content for a plurality of variations of a virtual        reality or augmented reality training session, the variations        including a least one of a training complexity and a training        pace;    -   serving content, associated with the training session, to        virtual reality or augmented reality client device;    -   adapting the flow of served content based on a metric of at        least one cognitive mental state of a user of the client device        that is based on biometric data of the user, where the at least        one cognitive mental state is associated with a learning        efficacy; and    -   the adapting being predictively selected to maintain the virtual        reality session within a selected range of the at least one        metric.

Example 22

The computer implemented method of Example 21, wherein the biometricdata further comprises at least one of a heart rate, eye tracking data,and motion tracking data.

Example 23

The computer implemented method of Example 21, wherein the at least onemetric comprises a cognitive load.

Example 24

The computer implemented method of Example 21, wherein the at least onemetric further comprises at least one of an anxiety level, a motivationlevel, a focus level, and an attention level.

Example 25

The computer implemented method of Example 21, wherein the adapting istriggered based on at least one threshold of the at least one metric.

Example 26

The computer implemented method of Example 21, further comprisinggenerating calibrating thresholds of the at least one metric for theuser in a test phase.

Example 27

The computer implemented method of Example 21, wherein the test phasecomprises monitoring the biometric data during the test phase inresponse to training material presented via the virtual reality oraugmented reality.

Example 28

The computer implemented method of Example 21, wherein the trainingmaterial is presented in a virtual reality session or an augmentedreality session.

Example 29

The computer implemented method of Example 21, further comprisingutilizing a trained machine learning model to perform the classifyingand the adapting.

Example 30

The computer implemented method of Example 21, wherein the adaptingcomprises at least one of adapting a complexity of the learning session,adapting a pace of the learning session, and incorporating rest periodsin the learning session.

Example 31

A system comprising:

-   -   a data storage unit storing content for a plurality of        variations of a virtual reality or augmented reality training        session, the variations including a least one of a training        complexity and a training pace;    -   a content server to server content, associated with the training        session, to virtual reality or augmented reality client device;    -   the content server adapting the flow of served content based on        a metric of at least one cognitive mental state of a user of the        client device that is based on biometric data of the user, where        the at least one cognitive mental state is associated with a        learning efficacy; and    -   the adapting being predictively selected to maintain the virtual        reality session within a selected range of the at least one        metric.

Example 32

The system of Example 31, wherein the content server includes aclassifier to classify the biometric data into the at least one metricof at least one cognitive mental state associated with a learningefficacy.

Example 33

The system of Example 32, wherein the content server comprises a machinelearning model trained to predictively adapt the flow of content.

Example 34

The system of Example 31, wherein the biometric data further comprisesat least one of a heart rate, eye tracking data, and motion trackingdata.

Example 35

The system of Example 31, wherein the at least one metric comprises acognitive load.

Example 36

The system of Example 31, wherein the at least one metric furthercomprises at least one of an anxiety level, a motivation level, a focuslevel, and an attention level.

Example 37

The system of Example 31, wherein the adapting is triggered based on atleast one threshold of the at least one metric.

Example 38

The system of Example 31, further comprising calibrating thresholds ofthe at least one metric for the user in a test phase.

Example 39

The system of Example 31, wherein the test phase comprises monitoringthe biometric data during the test phase in response to trainingmaterial presented via the virtual reality or augmented reality.

Example 40

The system of Example 31, wherein the training material is presented ina virtual reality session or an augmented reality session.

Example 41

The system of Example 31, wherein the adapting comprises at least one ofadapting a complexity of the learning session, adapting a pace of thelearning session, and incorporating rest periods in the learning session

Example 42

A computer-implemented method of neuroadaptive virtual reality trainingcomprising:

-   -   monitoring a plurality of different types of biometric data of a        user during a virtual reality or augmented reality training        session;    -   generating, from a first type of biometric including        electroencephalogram (EEG) biometric data, a first metric of a        first cognitive mental attribute of the user associated with a        learning efficacy; and    -   generating, from a second type of biometric data, a second        metric of a second cognitive mental attribute of the user        associated with a learning efficiency;    -   utilizing the first metric and the second metric as feedback        regarding an overall cognitive state of the user in the training        session; and    -   adapting the virtual reality or augmented reality training        session based on the feedback.

Example 43

The computer implemented method of Example 42, wherein the second typeof biometric data comprises at least one of a heart rate, eye trackingdata, and motion tracking data.

Example 44

The computer implemented method of Example 42, wherein the first metriccomprises a cognitive load.

Example 45

The computer implemented method of Example 42, wherein the second metriccomprises at least one of an anxiety level, a motivation level, a focuslevel, and an attention level.

Example 46

The computer implemented method of Example 42, wherein the adapting istriggered based on at least one threshold of the first metric and thesecond metric.

Example 47

The computer implemented method of Example 42, further comprisingcalibrating thresholds of the at least one metric for the user in a testphase.

Example 48

The computer implemented method of Example 42, wherein the test phasecomprises monitoring the biometric data during the test phase inresponse to training material presented via the virtual reality oraugmented reality.

Example 49

The computer implemented method of Example 42, wherein the trainingmaterial is presented in a virtual reality session or an augmentedreality session.

Example 50

The computer implemented method of Example 42, further comprisingutilizing a trained machine learning model to perform the classifyingand the adapting.

Example 51

The computer implemented method of Example 42, wherein the adaptingcomprises at least one of adapting a complexity of the learning session,adapting a pace of the learning session, and incorporating rest periodsin the learning session.

Example 52

A system comprising:

-   -   a biometric data monitor to monitor biometric data of a user        during a virtual reality or augmented reality training session,        the biometric data including electroencephalogram (EEG) data and        at least one other type of biometric data;    -   a biometric data analyzer to generate a first type of cognitive        mental state metric associated with a learning efficacy from the        EEG data, and generate a second type of cognitive mental state        metric from a second type of biometric data, wherein the second        type of cognitive mental state metric is associated with a        learning efficacy; and    -   a predictive engine utilizing the first metric and the second        metric to issue commands to adapt adapting the content of a        training session.

Example 53

The system of Example 52, wherein the biometric data further comprisesat least one of a heart rate, eye tracking data, and motion trackingdata.

Example 54

The system of Example 52, wherein the at least one metric comprises acognitive load.

Example 55

The system of Example 52, wherein the at least one metric furthercomprises at least one of an anxiety level, a motivation level, a focuslevel, and an attention level.

Example 56

The system of Example 52, wherein the adapting is triggered based on atleast one threshold of the at least one metric.

Example 57

The system of Example 52, further comprising calibrating thresholds ofthe at least one metric for the user in a test phase.

Example 58

The system of Example 52, wherein the test phase comprises monitoringthe biometric data during the test phase in response to trainingmaterial presented via the virtual reality or augmented reality.

Example 59

The system of Example 52, wherein the training material is presented ina virtual reality session or an augmented reality session.

Example 60

The system of Example 52, further comprising utilizing a trained machinelearning model to perform the classifying and the adapting.

Example 61

The system of Example 52, wherein the adapting comprises at least one ofadapting a complexity of the learning session, adapting a pace of thelearning session, and incorporating rest periods in the learning session

Other embodiments of the invention will be apparent to those skilled inthe art from consideration of the specification and practice of theinvention disclosed herein. It is intended that the specification andexamples be considered as exemplary only, with a true scope and spiritof the invention being indicated by the following claims.

Other embodiments of the invention will be apparent to those skilled inthe art from consideration of the specification and practice of theinvention disclosed herein. It is intended that the specification andexamples be considered as exemplary only, with a true scope and spiritof the invention being indicated by the following claims.

It is appreciated that certain features of the invention, which are, forclarity, described in the context of separate embodiments, may also beprovided in combination in a single embodiment. Conversely, variousfeatures of the invention, which are, for brevity, described in thecontext of a single embodiment, may also be provided separately or inany suitable subcombination.

Although the invention has been described in conjunction with specificembodiments thereof, it is evident that many alternatives, modificationsand variations will be apparent to those skilled in the art.Accordingly, it is intended to embrace all such alternatives,modifications and variations that fall within the broad scope of theclaims.

Other embodiments of the invention will be apparent to those skilled inthe art from consideration of the specification and practice of theinvention disclosed herein. It is intended that the specification andexamples be considered as exemplary only, with a true scope and spiritof the invention being indicated by the claims.

Reference in the specification to “one embodiment”, “some embodiments”or “an embodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least some embodiments of the disclosed technologies. Theappearances of the phrase “in some embodiments” in various places in thespecification are not necessarily all referring to the same embodiment.

Some portions of the detailed descriptions above were presented in termsof processes and symbolic representations of operations on data bitswithin a computer memory. A process can generally be considered aself-consistent sequence of steps leading to a result. The steps mayinvolve physical manipulations of physical quantities. These quantitiestake the form of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. Thesesignals may be referred to as being in the form of bits, values,elements, symbols, characters, terms, numbers, or the like.

These and similar terms can be associated with the appropriate physicalquantities and can be considered labels applied to these quantities.Unless specifically stated otherwise as apparent from the priordiscussion, it is appreciated that throughout the description,discussions utilizing terms, for example “processing” or “computing” or“calculating” or “determining” or “displaying” or the like, may refer tothe action and processes of a computer system, or similar electroniccomputing device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system's registersand memories into other data similarly represented as physicalquantities within the computer system memories or registers or othersuch information storage, transmission or display devices.

The disclosed technologies may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, or it may include ageneral-purpose computer selectively activated or reconfigured by acomputer program stored in the computer.

The disclosed technologies can take the form of an entirely hardwareimplementation, an entirely software implementation or an implementationcontaining both software and hardware elements. In some implementations,the technology is implemented in software, which includes but is notlimited to firmware, resident software, microcode, etc.

Furthermore, the disclosed technologies can take the form of a computerprogram product accessible from a non-transitory computer-usable orcomputer-readable medium providing program code for use by or inconnection with a computer or any instruction execution system. For thepurposes of this description, a computer-usable or computer-readablemedium can be any apparatus that can contain, store, communicate,propagate, or transport the program for use by or in connection with theinstruction execution system, apparatus, or device.

A computing system or data processing system suitable for storing and/orexecuting program code will include at least one processor (e.g., ahardware processor) coupled directly or indirectly to memory elementsthrough a system bus. The memory elements can include local memoryemployed during actual execution of the program code, bulk storage, andcache memories which provide temporary storage of at least some programcode in order to reduce the number of times code must be retrieved frombulk storage during execution.

Input/output or I/O devices (including but not limited to keyboards,displays, pointing devices, etc.) can be coupled to the system eitherdirectly or through intervening I/O controllers.

Network adapters may also be coupled to the system to enable the dataprocessing system to become coupled to other data processing systems orremote printers or storage devices through intervening private or publicnetworks. Modems, cable modems and Ethernet cards are just a few of thecurrently available types of network adapters.

Finally, the processes and displays presented herein may not beinherently related to any particular computer or other apparatus.Various general-purpose systems may be used with programs in accordancewith the teachings herein, or it may prove convenient to construct amore specialized apparatus to perform the required method steps. Therequired structure for a variety of these systems will appear from thedescription below. In addition, the disclosed technologies were notdescribed with reference to any particular programming language. It willbe appreciated that a variety of programming languages may be used toimplement the teachings of the technologies as described herein.

The foregoing description of the implementations of the presenttechniques and technologies has been presented for the purposes ofillustration and description. It is not intended to be exhaustive or tolimit the present techniques and technologies to the precise formdisclosed. Many modifications and variations are possible in light ofthe above teaching. It is intended that the scope of the presenttechniques and technologies be limited not by this detailed description.The present techniques and technologies may be implemented in otherspecific forms without departing from the spirit or essentialcharacteristics thereof. Likewise, the particular naming and division ofthe modules, routines, features, attributes, methodologies and otheraspects are not mandatory or significant, and the mechanisms thatimplement the present techniques and technologies or its features mayhave different names, divisions and/or formats. Furthermore, themodules, routines, features, attributes, methodologies and other aspectsof the present technology can be implemented as software, hardware,firmware or any combination of the three. Also, wherever a component, anexample of which is a module, is implemented as software, the componentcan be implemented as a standalone program, as part of a larger program,as a plurality of separate programs, as a statically or dynamicallylinked library, as a kernel loadable module, as a device driver, and/orin every and any other way known now or in the future in computerprogramming. Additionally, the present techniques and technologies arein no way limited to implementation in any specific programminglanguage, or for any specific operating system or environment.Accordingly, the disclosure of the present techniques and technologiesis intended to be illustrative, but not limiting.

What is claimed is:
 1. A computer-implemented method comprising:monitoring real-time biometric data of a user during an educationalsession, the biometric data including electroencephalogram (EEG) data;generating from the real-time biometric data at least one cognitivemental state metric associated with a learning efficacy; and performingpredictive adaption of education materials of the educational sessionbased on values of the at least one cognitive mental state metric togenerate commands to proactively adapt the education materials selectedfor presentation to the user on a user display device, the commandsbeing selected to maintain the at least one cognitive mental statemetric within a desired range to maintain learning efficiency.
 2. Thecomputer-implemented method of claim 1, wherein the real-time biometricdata further comprises at least one of a heart rate, eye tracking data,and motion tracking data.
 3. The computer-implemented method of claim 1,wherein the at least one cognitive mental state metric comprises acognitive load.
 4. The computer-implemented method of claim 3, whereinthe at least one cognitive mental state metric further comprises atleast one of an anxiety level, a motivation level, a focus level, and anattention level.
 5. The computer-implemented method of claim 1, whereinthe adapting is triggered based on at least one threshold value of theat least one cognitive mental state metric.
 6. The computer-implementedmethod of claim 1, wherein the generating at least one cognitive statemetric comprises using a classifier to classify the real-time biometricdata into the at least one cognitive state metric.
 7. Thecomputer-implemented method of claim 1, wherein the performingpredictive adaption comprises utilizing a machine learning model trainedto proactively adapt the education materials to maintain the at leastone cognitive mental state metric within a desired range to maintainlearning efficiency.
 8. The computer-implemented method of claim 1,wherein the adapting is selected to maintain the at least one cognitivemental state within a selected range of values.
 9. Thecomputer-implemented method of claim 1, further comprising utilizing atrained machine learning model to maintain the at least one cognitivemental state metric within a desired range and prevent the developmentof a cognitive mental state deleterious to learning.
 10. Thecomputer-implemented method of claim 1, wherein the adapting comprisesat least one of adapting a complexity of the learning session, adaptinga pace of the learning session, and incorporating rest periods in thelearning session.
 11. The computer-implemented method of claim 1,wherein the user display device comprises a wearable device.
 12. Thecomputer-implemented method of claim 11, wherein the user display devicecomprises a headset.
 13. The computer-implemented method of claim 11,wherein the wearable device is an augmented reality headset or a virtualreality headset.
 14. The computer-implemented method of claim 11, whereuser display device is laptop computer or a tablet device.
 15. Thecomputer-implemented method of claim 1, wherein the user display deviceis a non-wearable device.
 16. A computer-implemented method comprising:monitoring real-time biometric data of a user during an educationalsession, the biometric data including electroencephalogram (EEG) data;generating from the real-time biometric data at least one cognitivemental state metric associated with a learning efficacy; and performingpredictive adaption of education materials presented on a user displaydevice based on values of the at least one cognitive mental state metricto proactively adapt the education materials to maintain the at leastone cognitive mental state metric within a desired range to maintainlearning efficiency and prevent the development of a deleterious mentalstate for learning.
 17. A system comprising: a biometric data monitor tomonitor real-time biometric data of a user during an educationalsession, the biometric data including electroencephalogram (EEG) data; apredictive engine to classify the biometric data into at least onemetric of at least one cognitive mental state associated with a learningefficacy and generate commands to proactively adapt education materialsto maintain the at least one cognitive mental state metric within adesired range to maintain learning efficiency; and a content server toserve content to a user computing device having a display, the contentserver receiving the commands generated by the predictive engine and inresponse adapting the education materials to be presented to the userduring the educational session.
 18. The system of claim 17, wherein thereal-time biometric data further comprises at least one of a heart rate,eye tracking data, and motion tracking data.
 19. The system of claim 17,wherein the at least one cognitive mental state metric comprises acognitive load.
 20. The system of claim 19, wherein the at least onecognitive mental state metric further comprises at least one of ananxiety level, a motivation level, a focus level, and an attentionlevel.
 21. The system of claim 17, wherein the adapting is triggeredbased on at least one threshold of the at least one cognitive mentalstate metric.
 22. The system of claim 17, wherein a classifier is usedto classify the real-time biometric data into the at least one cognitivestate metric.
 23. The system of claim 17, wherein the predictive enginecomprises a machine learning model trained to proactively adapt theeducation materials to maintain the at least one cognitive mental statemetric within a desired range to maintain learning efficiency.
 24. Thesystem of claim 17, wherein the adapting is selected to maintain the atleast one cognitive mental state within a selected range of values. 25.The system of claim 17, wherein the predictive engine proactively adaptsthe educational materials to maintain the at least one cognitive mentalstate metric within a desired range and prevent the development of acognitive mental state deleterious to learning.
 26. The system of claim17, wherein the adapting comprises at least one of adapting a complexityof the learning session, adapting a pace of the learning session, andincorporating rest periods in the learning session.
 27. The system ofclaim 17, wherein the user computing device having a display comprises awearable device.
 28. The system of claim 27, wherein the user computingdevice having a display comprises a headset.
 29. The system of claim 28,where the non-wearable device is a laptop computer or a tablet device.30. The system of claim 27, wherein the wearable device is an augmentedreality headset or a virtual reality headset.
 31. The system of claim17, wherein the user computing device having a display comprises anon-wearable device.